Repository navigation
fix(evidence): stop calling a limitation "fitted with" when it says "not fitted" - #1173
Merged
NandhaKishorM merged 1 commit intoOct 14, 2026
Conversation
…not fitted" `inspect_calibration` prefixed every recorded limitation with "fitted with recorded limitations: ", then quoted the issue text from `laya.train.calibration_report` -- whose sub-`MIN_TYPE_N` entry reads "not fitted: 5 calibration items, fewer than 10, so the temperature stays 1.0". One `laya-evals evidence` line therefore asserted the fit and denied it in the same breath, which is the one thing the module docstring promises it never does (assign only states derived from the upstream artifact). The prefix now says "limitations recorded: " and adds no claim of its own. `tests/test_evidence.py` pinned calibration states only -- it had no check on `reason` at all -- so the three limitation kinds' full reason strings are now asserted verbatim, along with the rule that a reason never starts with "fitted".
Owner
|
Merged in 0.4.7. The line asserted a fit and denied it in the same breath: "limitations recorded: " is the right shape because the reason's job here is to carry the upstream text, not to add a claim of its own. The three verbatim checks plus |
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya/evidence.py::inspect_calibrationprefixed every recorded calibration limitation with"fitted with recorded limitations: ", then quoted the limitation text straight after it. One ofthe texts it quotes reads
"not fitted: ...", so a single line oflaya-evals evidenceoutputasserted the fit and denied it.
laya/evidence.py+4/-1 (@@ -60,7 +60,10 @@) -- the prefix is now"limitations recorded: ",plus a three-line comment on why the prefix claims nothing.
tests/test_evidence.py+17 (@@ -52,6 +52,23 @@) -- four checks on the reason text, whichnothing in that file used to assert at all.
No state, exit code,
detailpayload or function signature changes. A type that wasINSUFFICIENTis
INSUFFICIENTbefore and after, with the sameitemsandissuesindetail.Why
The artifact being read is written by
laya/train.py::calibration_report, whose first issue bulletis "fewer than
MIN_TYPE_Nitems: the temperature was not fitted and stays 1.0", emitted as"not fitted: %d calibration items, fewer than %d, so the temperature stays 1.0".evidence.pywraps that string in a sentence that says the opposite. Two processes, the same config, output pasted
exactly as
evidence.format_summaryprinted it -- first from a tree extracted fromc73908d9(release 0.4.6), then from this branch (the process asserted on its own
laya/evidence.pypath beforeprinting, so each block is that tree's output):
Every
stateis identical, thenoulline is byte-identical, and so is the overall line. Thechoiceline is the bug:
INSUFFICIENTis right, and the words before the quoted limitation asserted theopposite of the words after it.
That matters more here than a wording nit usually would, because this module's own docstring states
the rule it breaks: "Every state it assigns is either derived directly from an upstream artifact or
marked UNKNOWN. It never manufactures a healthy-looking state from absent input." The prefix is the
one piece of the reason line that is not from the artifact, and for the sub-
MIN_TYPE_Ntype it isa claim the artifact explicitly refuses to make.
The prefix is safe to reword:
grep -rn "fitted with recorded limitations"overlaya/,tests/and
docs/onc73908d9matches onlylaya/evidence.py:63-- nothing parses it, no test assertedany reason text (
grep -c reason tests/test_evidence.pyis 0 on main), and no doc page shows it.docs/evals.mddocuments the command without quoting its reason lines.How it was verified
tests/test_evidence.pygoes 40 (all 40 checks passed, from a tree extracted fromc73908d9,laya/evidence.pyandtests/test_evidence.pybyte-identical across 0.4.4/0.4.5/0.4.6 --f88d4b8eand887e6a65) -> all 44 checks passed. Full gateall 87 suites passed, rc 0, its pytest leg500 passed, 1 skipped in 15.73s, and500 passed, 1 skippedon the extracted tree as well(16.69s) -- unchanged, as expected:
tests/test_evidence.pyis a script suite inscripts/test_suites.pyand the 500 counts the separatePYTEST_SUITESinvocation, so the number thischange moves is that file's own 40 -> 44. ruff rc 0, compileall rc 0, docs strict build
No issues foundrc 0in 1.44s. The evidence summary appears in no built page
(
grep -rn "inspect_calibration\|limitations recorded" docs/is empty on this branch), so the strictbuild is a regression check here rather than a doc update.
The four new checks are the three limitation kinds' full reason strings asserted verbatim --
below-
MIN_TYPE_N, below-CALIB_WARN_Nand clamped, named<kind> reason is the prefix plus the upstream text, verbatim-- plus one rule check,no limitation reason states a fit the artifact did not record, that no limitation reason starts with"fitted". Verbatim rather than substring, because the bug was in the prefix and anassertIn(issue_text)would have passed with any prefix.Mutation check,
tests/test_evidence.pyafter each edit,laya/evidence.pyrestored withgit checkout --between runs (branch committed first,git statusclean before each run):all 44 checks passed"fitted with recorded limitations: "restoredno limitation reason states a fit the artifact did not record, which is the only one of the four that reads the shape rather than the exact string"recorded limitations: "(harmless reword, no false claim)"limitations recorded upstream", dropping the quoted upstream textgot 'limitations recorded upstream', expected 'limitations recorded: <upstream text>'So the verbatim pins are what notice a silent reword, and the rule check is what would notice the
specific class of reword that re-asserts a fit.
Deconfliction: 49 open PRs as at 2026-10-12T13:12:34Z, and the whole open set's file lists pulled
and matched line by line -- none of them edits
laya/evidence.pyortests/test_evidence.py.Not fixed here, worth knowing
PRESENTbranch still reads"fitted with no recorded issues on %d items". That one isdefensible and deliberately left alone: for an artifact
calibration_reportactually wrote, anempty
issueslist withitems > 0can only mean at leastMIN_TYPE_Nitems, because thesub-floor case always records its own issue. So "fitted" follows from the writer's invariant
there, where in the
INSUFFICIENTcase it contradicted the writer's own sentence. A hand-writtenconfig with
{"items": 5, "issues": []}does get that claim, and the loop atc73908d9tests/test_evidence.py:72-75(lines 89-92 on this branch, below this change's insertion)deliberately pins the state for that shape without judging the count -- keeping the module at one
numeric source of truth, as feat(evals): evidence inspection over checkpoint and eval artifacts #964 intended.
{"laya_train_calibration": {}}yields overallINSUFFICIENTwith no reason at all, while amissing calibration block yields
UNKNOWNwith a reason. That asymmetry is a state question,not a wording one, and belongs to whoever owns the ladder.
Checklist
main(c73908d9, release 0.4.6)